Qualcomm AI Engine Direct - Observer Fix and remove unused passes#6225
Qualcomm AI Engine Direct - Observer Fix and remove unused passes#6225winskuo-quic wants to merge 1 commit intopytorch:mainfrom
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6225
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit acc6f6d with merge base 1f2b9aa ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
Hi @cccclai,
|
|
Hi, thanks for sending the fix PR! We discussed internally regarding the index put node (#4627), and realized that these non-compute node ideally shouldn't be annotated, because they just move data around, and didn't do actual compute. Annotating them may cause a little bit regression (maybe not too much), but ideally, we can annotate computation operator only. I think it's better to have this PR in, given that some model accuracy drops to 0, but still better to resolve the above issue, and hopefully it helps the index put node issue |
cccclai
left a comment
cccclai
left a comment
There was a problem hiding this comment.
Left the comment in the github pr
|
@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Thanks for reviewing and the feedback! |
3 participants
Summary
ConvertToLinear()is redundant inqnn_preprocess.pysince this pass is already called inexecutorch/backends/qualcomm/utils/utils.pySome models are experiencing a significant drop in accuracy, with a few models having 0% accuracy. Adding new conditions to perform requantization and change ptq_per_channel_quant_config's IO from MinMaxObserver to MovingAverageMinMaxObserver to resolve the issue.
dtypeandis_dynamic. After this change, it checks for more attributes such asscale,zero_point, etc. This causes some nodes having an extra pair of QDQ nodes. As shown in the image below, there are 2 pairs of QDQ nodes after the PyTorch PR, and these 2 pairs of QDQ nodes have different scale and offset. For QNN lowering process, node will only save the quant info right after the node output. For example,catop below will usequantize_per_tensor_default_18's scale and offset as the node's quant attribute, and all other quant and dequant nodes will be ignored.This causes an accuracy drop, but by inserting a requantize node, we can see an improvement in accuracy for most models. Taking inceptionv3 as an example, the average top1 accuracy 0%->~75%. I have checked a couple other models and see accuracy either stays the same or have improvements.
I have also provided the option for users to skip this requant optimization if they preferred not to use it.
Before:
After
After the above change, it seems like there is an inference speed drop due to requantization. By switching to MovingAverageMinMaxObserver, I observed an improvement in inference speed for some models such as inceptionv3.